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(I . A system for accessing a substantially comprehensive record of an 
immediate environment of a user, comprising: 

a substantially continuous record of information from a plurality of 
time-correlated input data streams; 

means for specifying a query into the record; and 
means for displaying a result of a query made using said means for 
specifying a query. 



2. The system of claim 1 comprising: 

the input data streams are taken from a set consisting of video data, audio 
data, data representativespf the user's location and changes thereof, data 
representative of measurements of the user's physical state, and data 
representative of local ambien\environment conditions of said immediate 
environment, and current and reference standard time. 



15 3. The system of claim 1 , further domprising: 
means for collecting said record. 



20 



4. The system of claim 1 comprising: 

the plurality of time-correlated input data Streams includes at least one 
annotation data stream associated with at least one^of said time-correlated input 
streams. 
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The system of claim 4 comprising: 

the annotation data stream is from a set consisting of transcription of 
speech, information representative of people present on camera or speaking, 
information representative of location, information representative of user activities, 
informationyepresentative of aspects of video and audio input not directly input by 
said collecting\means, information representative of the user's biometric or 
emotional state^^nd information representative of other events of said immediate 
environment not directly input by said collecting means. 

6. The system of claim 4, further comprising: 
means for providin^vsaid annotation data stream. 

7. The system of claim 6 comprising: 

the means for providing said annotation data stream is an input device by 
which said user generates annotations. 

8. The system of claim 6 comprising: 
the means for providing said annotation data stream is an automatic input 

device wherein annotation is implemented based on other data in the record. 



9. The system of claim 8 wherein the means 6f providing said annotation data 
stream is triggered by detection that data in at least tme of said time-correlated 
input data streams matches a predetermined pattern. 
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1 6^ The system of claim 9, further comprising: 

\ means for specifying at least one said predetermined pattern and an 
annotation representative thereof to insert in the record when the predetermined 
pattern detected. 



1 1 . The system of claim 6 comprising: 

the meaner providing said annotation data stream includes means for 
using a remote data\source. 

\ 

12. The system of clait^ 1 comprising: 

\ 

the means for displaying a result includes means for presenting a 
time-bounded segment of at least one of said time-correlated input streams from 
the record. \ 



13. The system of claim 12 comprising: 
said means for presenting provides a presentation of a time-bounded 

segment of a plurality of said streams from \ie record wherein the presentations are 
overlaid. 

14. The system of claim 13, wherein one of said Streams is a video data stream, 
comprising: 

overlaying the video data stream with subtitles visible during video playback. 
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15\ The system of claim 13, wherein one of said streams is a video data stream, 
comprising: 

overlaying the video data stream by inserting word balloons visible during 
video playback. 

16. The system of claim 4 comprising: 

the means rpr displaying a result of a query includes means for presenting a 
set of annotations. 
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17. The system of claini 4 comprising: 

the means for specifying a query includes means for identifying a target 
time-bounded segment of the record. 

1 8. The system of claim 1 7 comprising: 

the means for identifying includes\means for finding a segment based on the 
attributes contained in the record during the. segment and means for specifying a 
set of attributes of interest for testing whether\he segment matches. 
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19. The system of claim 18, further comprising: 

said set of attributes includes a similarity metridv wherein the means for 
identifying identifies the target time-bounded segment based on strength of 
similarity under the similarity metric to a second segment; aqd 

means of identifying the second segment. 



20. The system of claim 19 wherein the second segment is an interval of time 
immediately prior to the present. 

21 . Tne system of claim 1 , further comprising: 

means for identifying a plurality of users and for proving system use 
authorization tor the users . 

22. The system otalaim 21 comprising: 

the means for displaying requires proof of authorization via said means for 
identifying a plurality of usqts and for proving system use authorization before 
displaying data from the record. 

23. The system of claim 21 , further comprising: 

means for identifying a subserspf the record and for providing authorization 
for viewing the subset to another persori or system. 

24. The system of claim 1 , further comprising: 

means of extracting a subset of the record and transmitting an extracted 
said subset to a secondary system or a device compatible with the system. 

25. The system of claim 1 , further comprising: \ 
means for compressing the record by deleting information. 
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The system of claim 25 comprising: 

the means for compressing the record includes a level-of-user-interest 
metric\deleting information based upon identifying uninteresting information in the 
record in accordance with said level-of-user-interest metric. 




27. The system of claim 26 wherein the interestingness metric consists of one or 
more criteria including changes in annotation, biometrically-determined interest or 
excitement level, frarr\e-to-frame video change, overall similarity between a 
selected segment of said time-correlated input data streams and segments which 
precede said selected segment. 



28. The system of claim 26 wherein the deleting information is based upon 
reducing the frame rate of a video apta stream of said time-correlated input data 
streams . 

29. The system of claim 26 wherein the plurality of time-correlated input data 
streams includes at least one annotation data\tream associated with at least one 
specified one of said time-correlated input streams and wherein the deleting of 
information is based upon discarding input stream d\ta and retaining annotations 
of said at least one annotation data stream. 



30. The system of claim 26, wherein the record comprises^a portable part and 
an archival part, the system further comprising: 

means for transmitting a subset of the record wherein delving information 
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is based upon transmitting information to be deleted from the portable part to the 
archival part. 



31 . A portable device for capturing a substantially comprehensive record of an 
immediate et^vironment of a user, comprising: 
a portable housing; 

associate\with said housing, one or more data collection devices; and 
integrated witji said housing, a time-keeping device, a data storage device, 



and a programmable device for correlating all data captured by the data collection 
devices based upon timevreported by the time-keeping device and for storing so- 
correlated data on the storage device. 



32. The device of claim 31 , further comprising: 
associated with the housing^at least one input device, the programmable 

device further operable to use input from each said input device to record 
time-stamped annotations in the recordN 

33. The device of claim 31 comprising: 
the programmable device having prograrh code for determining a segment 

of the record to be uninteresting and to compress ijie record by deleting or 
degrading segments so-determined. 



34. The device of claim 31 , further comprising: 

a communications port for transmitting a subset of tl\e record stored on the 
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data storage device. 

35. V wearable device for querying a substantially comprehensive record of an 
immediate environment of a user, comprising: 

a portable housing; 

associated with said housing, at least one data output device, and at least 
one input device for specifying data queries; and 

integrated with said housing, a data storage device containing a data record 
and a programmable atevice for accepting queries from the at least one input 
device, for identifying sections of the record based on the queries, and for 
displaying the results of thexaueries on the at least one output device. 

36. The device of claim 35, further comprising: 

associated with the housing, an identification device for identifying a user as 
authorized to make said queries. \ 

37. A system for data collection comprising: 
a portable housing; and \ 

interconnected within said housing, data collection means for capturing data 
representative of an immediate vicinity of the system, integration means for 
correlating said autobiographical data, and memory means for storing said 
autobiographical data. \ 
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38\ The system as set forth in claim 37, said data collection means comprising: 

audio-video devices for capturing photo-images and accompanying sound in 
the immediate vicinity of said system. 



39. The system as set forth in claim 38, said audio-video devices further 
comprising: 



• system as 

' : \ 



a camera oata processor, and 

a remote camera lens for transmitting photo-images to said camera data 
processor. 

40. The system as set f<\th in claim 38, said audio-video devices further 
comprising: 

an audio signal processoK and 

a remote microphone for transmitting audio signals to said audio signal 
processor. 

41. The system as set forth in claim 37\said data collection means comprising: 
global positioning system (GPS) for capturing GPS data of said system. 



42. The system as set forth in claim 37, said data collection means comprising: 

connected to an end-user of the system, biom^tric sensors for capturing 
system user biometric readings; 
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43. \"The system as set forth in claim 37, said data collection means comprising: 

environmental sensors for capturing environmental condition data in the 
immediate vicinity of said system. 

44. The sVstem as set forth in claim 37, said data collection means further 
comprising: \ 

connectea\to said integration means, at least one direct data input port for 
receiving digital dara from a remote source. 

45. The system as s^t forth in claim 37, further comprising: 
feedback means for controlling said system and for monitoring data 

collection in real time. \ 

46. The system as set forth invclaim 45, said feedback means further 
comprising: \ 

a heads-up display for monitoring data which comprises perceivable 
stimulus data. \ 

47. The system as set forth in claim 45, sand feedback means further 
comprising: \ 

an electromechanical display and command entry panel. 

48. The system as set forth in claim 45, said feedback means further 
comprising: \ 
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\ an audio transducer for audibly transmitting sound data collected by said 
data collection means. 

49. The system as set forth in claim 37, said integration means further 
comprising: \ 

a programmable device including routines for real-time integrating of the 
autobiographical datk storing the autobiographical data, and downloading 
autobiographical data tc* an off-board data storage apparatus. 

50. The system as set forth in claim 45, said feedback means further 
comprising: \ 

means for retrieving autobiographical data selectively based on 
predetermined search parameters.X 

51 . The system as set forth in claim 4V further comprising: 
routines for editing 323, 325 the autobiographical data. 

52. A process for generating autobiographicaPdata, the method comprising: 
providing an integrated apparatus for collecting data representative of 

perceptual stimuli in the immediate vicinity of a persork 
continuously collecting said data; \ 
in real-time, integrating said data in accordance withVredetermined 
relational characteristics of said perceptual stimuli into a content retrievable data 
collection; and \ 
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storing said data collection in a memory. 

53. Tfre process as set forth in claim 52, said collecting further comprising: 
receiving time-correlated input data streams representative of real-time 

information related to visual stimuli, audio stimuli, geophysical position, local 
environmenta\conditions, and autobiographical biophysical condition of said 
person. 

54. The process a^set forth in claim 53 wherein said data streams include 
annotation data. 

55. The process as set forth in claim 54 wherein said annotation data is 
selected from a group including transcription of speech data, a digital marker 
related to at least one of said dataVtreams, a digital marker related to an interest 
level of at least one of said streams, \digital marker related to biometric condition 
of said person. 

56. The process as set forth in claim 54 w\erein said annotation data is 
manually provided by said person. 



57. The process as set forth in claim 54 wherein skid annotation data is 
automatically generated based on other data in a current one of said data streams. 
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The process as set forth in claim 57 wherein said annotation data is 
automatically generated based upon detection that said current one of said data 
streams\matches a pattern of data in said memory. 

59. The process as set forth in claim 54 wherein said annotation data is 
provided from a\remote data source. 

60. The process as set forth in claim 54 further comprising: 
displaying data from said collection. 

61 . The process as set forth in claim 60, said displaying further comprising: 
presenting a time-bounded segment of at least one stream of said data. 

62. The process as set forth in cfeum 61 said presenting further comprising: 
overlaying a plurality of streamsspf said data. 

63. The process as set forth in claim 62 Wierein said annotation data for 
overlaying of a video data stream and a transcription data stream is by inserting 
subtitles in the video data stream. \ 

64. The process as set forth in claim 62 wherein annotation data for overlaying 
of a video data stream and a transcription data stream ik by inserting a word 
balloon in the video data stream. \ 
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65\ The process as set forth in claim 60, said displaying further comprising: 
.presenting a set of said annotations as a response to specifying a query. 

66. The process as set forth in claim 65, said specifying further comprising: 
identifying a target time-bounded segment of said collection. 

67. The process as set forth in claim 66, said identifying further comprising: 
specifying a^set of attributes, and 

finding the target time-bounded segment based on said attributes contained 
in the collection during tne target time-bounded segment. 

68. The process as set forth in claim 66, said identifying further comprising: 
specifying a similarity metric related to a first time-bounded segment, and 
finding the target time-bounded segment based on a said similarity metric. 

69. The process as set forth in claim (S8 wherein said first time-bounded 
segment is immediately prior to present tim* 

70. The process as set forth in claim 54, furthW comprising: 
on-demand, compressing said data collection 



71 . The process as set forth in claim 70, said compressing further comprising: 
specifying deletion parameters, and 
deleting information based upon said parameters. 
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72. y The process as set forth in claim 71 wherein said deletion parameters 
includfe a parameter representative of data level of interest of the person. 



73. The process as set forth in claim 72 wherein parameter representative of 
of interest of i 



data level 



the person is selected from a group comprising changes in 



annotation, biometrically-determined interest or excitement level, frame-to-frame 
video change, andNoverall similarity between a current segment and earlier time- 
bounded segments. 



73. The process as set forth in claim 71, said deleting further comprising: 
deleting specified time-bounded segments while retaining the annotations 

related thereto. 

74. The process as set forth in cla)m 52 wherein said memory includes a 
portable memory associated with said collecting and an archival memory and said 
storing further comprises transmitting a sublet of said collection from said portable 
memory to said archival memory. 



75. Surveillance apparatus comprising: 
a portable housing; and 

integrated with said housing, a camera, an audio Recorder, a GPS, a data 
memory, and a programmable device for integrating all date captured by said 
camera, audio recorder, and GPS into an integrated, contentVetrievable format 
and for storing and retrieving data so formatted from said memqry. 

^Docket No.:10010971-1 

-39- 



76\ The apparatus as set forth in claim 75, further comprising: 

\ associated with said housing a biometric monitor for connecting to a subject, 
and \ 

said programmable device including routines for integrating data collected 
by said monitor with said all data captured by said camera, audio recorder and 
GPS. \ 

77. The apparatus as set forth in claim 75, further comprising: 

associated witn\?aid housing a local environmental condition data recorder, 
and, \ 

said programmable itevice including routines for integrating data collected 
by said monitor with said all dsita captured by said camera, audio recorder and 
GPS. \ 

78. The system as set forth in clairn 75, said programmable device further 
comprising: \ 

program code for analyzing a segment of said data based on at least one 
predetermined parameter and for compressing^the segment based on said 
analyzing. \ 

79. The system as set forth in claim 75, further comprising: 

a mechanism for transmitting said captured data from said system. 
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The system as set forth in claim 75, further comprising: 

associated with said housing, a mechanism for displaying said captured 



data. 



81 . The system as set fortKln^claim 80, said programmable device further 
comprising: 

program code for accepting a data qttejy, for identifying segments of the 
captured data based on said query, and for displacing said segments on said 
mechanism for displaying. 
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